The past decade has been characterized by significant development and research in AR, AR content may, include for example, of 3D virtual models, which can be created using modeling software (application). According to prior art, the virtual content (such as AR content) is usually stored within a database (library) or model file, and is fetched by the AR system for rendering. Hence, according to the prior art, the AR content is digitally stored, and any interaction with it, such as editing, requires using dedicated computer software.
Content libraries/databases in AR systems are commonly maintained either locally on the augmenting device, or remotely on a database server. Updating local libraries on user devices is a relatively complicated task (especially if the devices are distributed between a large number of users). On the other hand, updating remote libraries (databases) is a relatively simple and centralized task, yet it requires connecting to a data network, such as the Internet or a cellular network. For example, it is supposed that a user points a camera of his cellular phone, on which an advertising AR application is installed, to an image printed in a newspaper. Then, the user observes a virtual scene containing one or more AR models, which are fetched from a remote database, superimposed over the printed image. For this purpose, according to the prior art, in order to download the AR content the user can point his cellular phone camera to a pattern or a barcode in the newspaper that encodes an index to the corresponding AR content to be downloaded. After that, the advertising AR application (installed on his cellular phone) connects to a remote database for downloading the AR, content according to the index. Therefore, according to the above, a user takes an active role in retrieving the AR contents. In addition the user may not be able to modify the content since the models compositing the AR scene are hard-coded in the database. The interaction with the models is also hard-coded in the database or even in the application itself.
Further, according to the prior art, the AR content can be digitally created by dedicated 3D modeling software. Such virtual content creation process usually requires relatively highly skilled content creators. Also, the 3D models have to be designed, as do their behaviors (for example, their time-dependent animation).
Also, according to the prior art, barcodes are widely used as a vision-based mechanism for identification, and can contain relatively complicated information. For example, the Gameboy® product of the Nintendo® company (located in United States), uses electronic cards with barcodes, wherein a complete game can be encoded within a barcode. Each individual electronic card has enough memory space to digitally encode a complete mini-game. Barcodes can also be used to identify a remote source of content (for example, barcodes can encode web page addresses). When the barcode is scanned by means of a cellular phone, the remote source of content is accessed over the cellular network. Although barcodes usually appear as a collection of black dots or lines, images can be incorporated into the barcodes in order to make them visually pleasant, as shown on FIG. 1A (incorporating photo 105 of a man's face). The barcode can be, for example, a dynamic 4D barcode, allowing the transmission of data to a mobile device (e.g., cellular phone 120) by providing a stream of barcode images, as illustrated in FIG. 1B.
Moreover, according to the prior art, in computer vision-based AR systems, visual markers are commonly used. The markers are usually used for: (1) identifying the AR content that should be augmented in a real-world scene; and (2) performing registration, thereby determining camera location relative to real objects within the real-world scene. For example, ARToolKit®, a software library for building AR applications developed and maintained by Human Interface Technology (HIT) Laboratory, University of Washington, USA, uses square fiducial markers (as illustrated on FIG. 1C). In addition, in the AR PlayStation® game titled “The Eye of Judgment” (http://www.eyeofjudgment.com/), cards are used to identify models and actions that can be performed: the cards are placed on a tracked board and their appearance reflects the content they identify.
It should also be noted that according to the prior art, several methods exist for interpreting relatively rich content from symbols, and more generally, from visual languages. Some visual languages are understandable to people without any computer system. For example, Pictography is the expression of words and ideas through standard languages that omit unnecessary details, e.g., icons, road signs, and Chinese characters. Visual languages are also common in cartography: non-spatial features are commonly described by legends, and spatial information, such as terrain topography or temperature is commonly described using color scales. It should be noted that a sketched 3D drawing (e.g., a perspective or isometric sketch of a cube) is also based on a visual language that is understandable to people; people who see such a perspective sketch of the cube, can relatively easily interpret it as the corresponding 3D object.
In addition, it should be noted that different approaches have been described in the literature for analyzing graphical annotations. For example, mathematical and physical laws can be automatically interpreted.
Therefore, there is a need in the prior art to provide AR method and system, wherein AR content is embedded within an image (e.g., a sketch) by eliminating the need for downloading the AR content from a local or remote library/database, and allowing interaction with the the embedded content by hand sketching on the image; i.e., there is a need to provide a convenient and efficient way to generate AR content without maintaining a data content library/database, thereby extracting virtual model geometry, texture and behavior substantially directly from the image. For a better understanding of, and interaction with the embedded content there is a need for the content to employ dual perception. Dual perception means that the the embedded AR content is visually understandable also without using any AR system. Furthermore, there is a need to provide a way to interact with AR content to be augmented by enabling editing the AR content (e.g., by sketching), while displaying it to a user, in substantially real-time.